{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de765f16-341b-4dc3-b59d-b16437c7e050",
   "metadata": {},
   "source": [
    "## RAG - Structurizr Language QA\n",
    "\n",
    "- UnstructuredMarkdownLoader\n",
    "- MarkdownTextSplitter\n",
    "- OllamaEmbeddings\n",
    "- DocArrayInMemorySearch\n",
    "- OllamaLLM\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "16195893-50e1-426d-bec2-6706c24e44f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install unstructured==0.16.5 markdown==3.7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "964deaf8-c8f9-4310-b70b-ec384ecb3821",
   "metadata": {},
   "source": [
    "## Create Loader to load language-reference.md from local file system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e9544758-d27a-40f9-bcbf-3f5c881957f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredMarkdownLoader\n",
    "markdownLoader = UnstructuredMarkdownLoader(\"structurizr/dsl/docs/language-reference.md\")\n",
    "docs = markdownLoader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a728508f-9769-4979-92e0-30151ac96164",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.59 ms, sys: 2.19 ms, total: 8.78 ms\n",
      "Wall time: 11 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from langchain.text_splitter import MarkdownTextSplitter\n",
    "\n",
    "text_splitter = MarkdownTextSplitter(chunk_size=200, chunk_overlap=50)\n",
    "\n",
    "texts = text_splitter.split_documents(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5bcab33-d0ec-4845-b4af-070905b744ea",
   "metadata": {},
   "source": [
    "## Create Embedding Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9b1eb0b2-d6de-4530-950c-bb4336c2b3c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_ollama import OllamaEmbeddings\n",
    "\n",
    "ollama_url = \"http://ollama:11434\"\n",
    "\n",
    "model_name = \"nomic-embed-text:v1.5\"\n",
    "\n",
    "embeddings = OllamaEmbeddings(\n",
    "    base_url = ollama_url,\n",
    "    model=model_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a1b3ca9-49cb-4f38-b9b3-5954f9e00fbd",
   "metadata": {},
   "source": [
    "## Create VectorStore and start embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "77569ae3-3398-4b12-b95e-c0bb73bd5cd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.11/site-packages/pydantic/_migration.py:283: UserWarning: `pydantic.error_wrappers:ValidationError` has been moved to `pydantic:ValidationError`.\n",
      "  warnings.warn(f'`{import_path}` has been moved to `{new_location}`.')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 83.3 ms, sys: 11.1 ms, total: 94.4 ms\n",
      "Wall time: 1min 26s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from langchain_community.vectorstores import DocArrayInMemorySearch\n",
    "\n",
    "vectorstore = DocArrayInMemorySearch.from_documents(texts, embeddings)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c178cf6f-e4fc-41a0-aad3-c85e0b7e863a",
   "metadata": {},
   "source": [
    "## Create Retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cd4a179d-be7e-459a-8440-25970a1d386e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.runnables import RunnableParallel, RunnablePassthrough\n",
    "\n",
    "retrieval = RunnableParallel(\n",
    "    {\"context\": vectorstore.as_retriever(), \"input\": RunnablePassthrough()}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ea11a09-2371-4c4b-bf2e-2dbdd36d6b90",
   "metadata": {},
   "source": [
    "## Create Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4199b252-1cd1-48b9-843d-50277261d87d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"You are a software architect with knowledage in structurizr helping software architecture modeling and documenation in the following context: {context}\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "967602d3-45cc-4c54-b10c-dc649c794bc2",
   "metadata": {},
   "source": [
    "## Create LLM for QA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e4ff3fad-6476-41ef-9045-8edde6b0eb22",
   "metadata": {},
   "outputs": [],
   "source": [
    "ollama_url = \"http://ollama:11434\"\n",
    "# ollama_url = \"http://localhost:11434\"\n",
    "\n",
    "model_name = \"mistral:v0.3\"\n",
    "\n",
    "from langchain_ollama import OllamaLLM\n",
    "\n",
    "llm = OllamaLLM(base_url=ollama_url,model=model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cc29c64-9162-4d67-9d95-f7edf379523d",
   "metadata": {},
   "source": [
    "## Create OutputParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e9f67d85-f476-4298-938b-53d3fc0cd2f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "output_parser = StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8852ed3d-5254-4856-ac3d-6043f6fecdb6",
   "metadata": {},
   "source": [
    "## Create Chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e9c9cc22-3e02-4e95-a694-73900b89b1df",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = retrieval | prompt | llm | output_parser"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c1af482-015d-4ff0-bb4b-477f3b9f7680",
   "metadata": {},
   "source": [
    "## Invoke Chain with keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d8d62baf-b7bc-4dd9-a8e3-30dd1b3e0ed2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " In the context of Structurizr, a workspace is the top-level language construct and serves as a wrapper for the model and views. A workspace can optionally be named and described. Here's an example of its basic syntax:\n",
      "\n",
      "```\n",
      "workspace [name] [description] { ... }\n",
      "```\n",
      "\n",
      "A workspace can also extend another workspace, which allows you to add more elements, relationships, views, etc., to the extended workspace. Here's an example of extending a workspace:\n",
      "\n",
      "```\n",
      "workspace extends <file|url> { ... }\n",
      "```\n",
      "\n",
      "By extending another workspace, all identifiers defined in that workspace are available for use in the extended workspace. This feature allows for the reuse and organization of common elements across your models.\n",
      "\n",
      "CPU times: user 794 ms, sys: 20.7 ms, total: 815 ms\n",
      "Wall time: 1min 2s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for chunk in chain.stream(\"workspace\"):\n",
    "    print(chunk, end=\"\", flush=True)\n",
    "\n",
    "print(\"\\n\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b71ec07d-b779-43a2-ad70-537f46427529",
   "metadata": {},
   "source": [
    "### Analysis\n",
    "\n",
    "The following content is from the section \"### workspace\" in [structurizr/dsl/docs/language-reference.md](structurizr/dsl/docs/language-reference.md), the generation from LLM is very relevant to the orignal content.\n",
    "\n",
    "#### workspace\n",
    "`workspace` is the top level language construct, and the wrapper for the [model](#model) and [views](#views). A workspace can optionally be given a name and description.\n",
    "\n",
    "```\n",
    "workspace [name] [description] {\n",
    "    ...\n",
    "}\n",
    "```\n",
    "\n",
    "A workspace can also extend another workspace, to add more elements, relationships, views, etc to it.\n",
    "\n",
    "```\n",
    "workspace extends <file|url> {\n",
    "    ...\n",
    "}\n",
    "```\n",
    "\n",
    "The base workspace can either be referenced using a local DSL/JSON file, or a remote (via a HTTPS URL) DSL/JSON file.\n",
    "When extending a DSL-based workspace, all of the identifiers defined in that workspace are available to use in the extended workspace.\n",
    "\n",
    "Permitted children:\n",
    "\n",
    "- name <name>\n",
    "- description <description>\n",
    "- [properties](#properties)\n",
    "- [!docs](#documentation)\n",
    "- [!adrs](#architecture-decision-records-adrs)\n",
    "- [!identifiers](#identifier-scope)\n",
    "- [!impliedRelationships](#impliedrelationships)\n",
    "- [model](#model)\n",
    "- [views](#views)\n",
    "- [configuration](#configuration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "87692444-00ef-4cd6-a17c-02ea3e30b343",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The `softwareSystem` keyword in the context of Structurizr is used to define a software system. Here's a brief explanation:\n",
      "\n",
      "- A software system (e.g., `Software System`, `Software System 2`) represents a complex, distributed or monolithic software application that is being modeled using Structurizr. It can have a name, description, and tags for better identification and understanding.\n",
      "\n",
      "- The API container within a software system (e.g., `api = container \"API\"`) defines an important component of the software system, which is likely to be an interface for external systems or users to interact with the software system.\n",
      "\n",
      "Additionally, you mentioned some other terms like `url`, `properties`, and `perspectives`. However, these terms seem not to be directly related to the definition of a software system but rather general concepts within Structurizr's modeling language:\n",
      "\n",
      "- `url`: A reference or link to an external resource.\n",
      "- `properties`: Attributes or characteristics of a model element.\n",
      "- `perspectives`: Different views or representations of a model, focusing on various aspects like architecture, design, development, testing, and deployment.\n",
      "- The `-> (relationship)` indicates a relationship between two elements in the model.\n",
      "\n",
      "For more detailed information about these terms, I would recommend consulting the official Structurizr documentation.\n",
      "\n",
      "CPU times: user 794 ms, sys: 31.4 ms, total: 825 ms\n",
      "Wall time: 1min 12s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for chunk in chain.stream(\"softwareSystem\"):\n",
    "    print(chunk, end=\"\", flush=True)\n",
    "\n",
    "print(\"\\n\", flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fc09d93c-7e79-424d-a4bd-578a37c5c26c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The syntax for `softwareSystem` in the provided context is as follows:\n",
      "\n",
      "```\n",
      "softwareSystem <name> [description] [tags] { ... }\n",
      "```\n",
      "\n",
      "In this syntax:\n",
      "- `<name>` is the unique name given to the software system.\n",
      "- `[description]` (optional) is a brief description of the software system.\n",
      "- `[tags]` (optional) are keywords associated with the software system.\n",
      "- The `{ ... }` section represents the properties, perspectives, and relationships of the software system.\n",
      "\n",
      "CPU times: user 637 ms, sys: 8.18 ms, total: 646 ms\n",
      "Wall time: 21.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for chunk in chain.stream(\"what is the syntax of softwareSystem\"):\n",
    "    print(chunk, end=\"\", flush=True)\n",
    "\n",
    "print(\"\\n\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f703242e-2f70-4efd-b533-14002eab81fc",
   "metadata": {},
   "source": [
    "### Analysis\n",
    "\n",
    "The following content is from the section \"### softwareSystem\" in [structurizr/dsl/docs/language-reference.md](structurizr/dsl/docs/language-reference.md), the generation from LLM is very relevant to the orignal content.\n",
    "\n",
    "#### softwareSystem\n",
    "The `softwareSystem` keyword defines a software system.\n",
    "\n",
    "```\n",
    "softwareSystem <name> [description] [tags] {\n",
    "    ...\n",
    "}\n",
    "```\n",
    "\n",
    "The following tags are added by default:\n",
    "\n",
    "- `Element`\n",
    "- `Software System`\n",
    "\n",
    "Permitted children:\n",
    "\n",
    "- [!docs](#documentation)\n",
    "- [!adrs](#architecture-decision-records-adrs)\n",
    "- [group](#group)\n",
    "- [container](#container)\n",
    "- [description](#description)\n",
    "- [tags](#tags)\n",
    "- [url](#url)\n",
    "- [properties](#properties)\n",
    "- [perspectives](#perspectives)\n",
    "- [-> (relationship)](#relationship)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ed2447c4-2c1e-4369-92ca-df6314dab182",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " In Structurizr, the syntax for defining a container in a software system is as follows:\n",
      "\n",
      "```\n",
      "container <name> [description] [technology] [tags] { ... }\n",
      "```\n",
      "\n",
      "Here's a breakdown of each part:\n",
      "\n",
      "- `<name>`: This is the name of the container. It's required and should be descriptive.\n",
      "- `[description]`: This is optional and provides additional information about the container.\n",
      "- `[technology]`: This is also optional and specifies the technology used in this container.\n",
      "- `[tags]`: This is also optional but helps to categorize the container for easier navigation. Tags are comma-separated.\n",
      "\n",
      "The `{ ... }` indicates that the content within these braces provides more details about the container, such as its properties, descriptions, relationships, and so on.\n",
      "\n",
      "You can also define a Container Instance, which represents an instance of a container within your software system:\n",
      "\n",
      "```\n",
      "containerInstance <identifier> [deploymentGroups] [tags] { ... }\n",
      "```\n",
      "\n",
      "- `<identifier>`: This is required and should uniquely identify the specific instance of the container.\n",
      "- `[deploymentGroups]`: This is optional and allows you to specify one or more deployment groups associated with this container instance. Deployment groups are comma-separated.\n",
      "- The rest of the syntax (tags, description, etc.) follows the same pattern as for a container.\n",
      "\n",
      "CPU times: user 775 ms, sys: 30 ms, total: 805 ms\n",
      "Wall time: 1min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for chunk in chain.stream(\"what is the syntax of Container\"):\n",
    "    print(chunk, end=\"\", flush=True)\n",
    "\n",
    "print(\"\\n\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26ec816a-a080-4b22-81fa-d0848a0c8473",
   "metadata": {},
   "source": [
    "### Analysis\n",
    "\n",
    "The following content is from the sections \"### Container\" and \"### containerInstance\" in [structurizr/dsl/docs/language-reference.md](structurizr/dsl/docs/language-reference.md), the generation from LLM is very relevant to the orignal content.\n",
    "\n",
    "#### container\n",
    "\n",
    "The `container` keyword defines a container, within a software system.\n",
    "\n",
    "```\n",
    "container <name> [description] [technology] [tags] {\n",
    "    ...\n",
    "}\n",
    "```\n",
    "\n",
    "The following tags are added by default:\n",
    "\n",
    "- `Element`\n",
    "- `Container`\n",
    "\n",
    "Permitted children:\n",
    "\n",
    "- [!docs](#documentation)\n",
    "- [!adrs](#architecture-decision-records-adrs)\n",
    "- [group](#group)\n",
    "- [component](#component)\n",
    "- [description](#description)\n",
    "- [technology](#technology)\n",
    "- [tags](#tags)\n",
    "- [url](#url)\n",
    "- [properties](#properties)\n",
    "- [perspectives](#perspectives)\n",
    "- [-> (relationship)](#relationship)\n",
    "\n",
    "#### containerInstance\n",
    "\n",
    "The `containerInstance` keyword defines an instance of the specified container that is deployed on the parent deployment node.\n",
    "\n",
    "```\n",
    "containerInstance <identifier> [deploymentGroups] [tags] {\n",
    "    ...\n",
    "}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
